Prune Diseased Branches to Get Healthy Trees! How to Find Erroneous Local Trees in a Treebank and Why It Matters

Annotated corpora are essential for training and testing algorithms in natural language processing (NLP), but even so-called gold-standard corpora contain a significant number of annotation errors (cf. Dickinson 2005, and references therein). For part-of-speech annotation, these errors have been shown to be problematic for both training and evaluation of NLP technology (van Halteren 2000; Padro and Marquez 1998; van Halteren et al. 2001; Květǒn and Oliva 2002). But only little work has been done on detecting errors in syntactic annotation (Ule and Simov 2004; Dickinson and Meurers 2003b, 2005), and the effect of the errors detected on the uses of such corpora has not been systematically explored. In this paper, we describe a new method for finding errors in treebanks and demonstrate the effect of such errors on NLP technology. Similar to the work in Dickinson and Meurers (2003b, 2005)—where multiple occurrences of the same string in identical contexts are found with varying labels—the approach presented here is based on the detection of inconsistencies; but instead of focusing on the consistent assignment of a label to a string, we here investigate the consistency of labeling within local trees. In section 2, we describe the new method, and we show the results of applying it to the Wall Street Journal corpus in section 3. After discussing ways to automatically identify individual erroneous rules in section 4, we show in section 5 that eliminating the detected errors from the training data of a probabilistic context-free grammar (PCFG) parser improves its performance.